Categorizing concept terms for game-based training in cognitive computing systems

ABSTRACT

A first list that includes a first set of one or more concept terms may be obtained. The first set of one or more concept terms may be candidates for being conceptually related to a seed concept term. Prior to the obtaining, a first client computing device may provide a first plurality of individual characters and the seed concept term to a first user using the first client computing device. The first client computing device may prompt the first user to generate the first set of one or more concept terms that are conceptually related to the seed concept term using one or more of the first plurality of individual characters. In response to the obtaining, one or more particular categories to which each of the first set of one or more concept terms belongs to may be determined.

BACKGROUND

The present disclosure relates to cognitive computing systems, and morespecifically to game-based training for cognitive computing systems.

Recent research has been directed to developing cognitive computingsystems (e.g., concept expansion systems, question answering (QA)systems, etc.). Cognitive computing systems may build knowledge andlearn (e.g., via training), understand natural language, reason, quicklyidentify new patterns, put content in context with confidence scores,analyze terms and interpret the terms' meanings, all of which mayultimately model human intelligence. For example, QA systems may bedesigned to receive input questions, analyze them, and return applicablecandidate answers. These systems may rely on natural languageprocessing, automated reasoning, machine learning, and other advancedtechniques. Using these techniques, QA systems may provide mechanismsfor searching large sources of content and analyzing the content withregard to a given input question in order to determine an answer to thequestion. In some QA systems this may take the form of hypothesisgeneration, scoring, and ranking in order to determine a final set ofone or more output answers.

SUMMARY

Embodiments of the present disclosure may include a method, a system,and a computer program product. A cognitive computing system may obtaina first list that includes a first set of one or more concept terms. Thefirst set of one or more concept terms may be transmitted from a firstclient computing device to the cognitive computing system. The first setof one or more concept terms may be candidates for being conceptuallyrelated to a seed concept term. Prior to the obtaining, the first clientcomputing device may provide a first plurality of individual charactersand the seed concept term to a first user using the first clientcomputing device. The first client computing device may prompt the firstuser to generate the first set of one or more concept terms that areconceptually related to the seed concept term using one or more of thefirst plurality of individual characters. The cognitive computing systemmay determine, in response to the obtaining, one or more particularcategories to which each of the first set of one or more concept termsbelongs to. Each of the particular categories may be a distinct classthat includes an identification. The identification may describes aspecific meaning for each of the first set of one or more concept terms.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings included in the present application are incorporated into,and form part of, the specification. They illustrate embodiments of thepresent disclosure and, along with the description, serve to explain theprinciples of the disclosure. The drawings are only illustrative of someembodiments and do not limit the disclosure.

FIG. 1 illustrates a block diagram of an example computing environment,consistent with embodiments of the present disclosure.

FIG. 2 illustrates a block diagram of a natural language processing(NLP) system usable to generate answer estimates to one or morequestions, expand seed concept terms, etc., consistent with embodimentsof the present disclosure.

FIG. 3 illustrates a block diagram of an example high level logicalarchitecture of a cognitive computing system, consistent withembodiments the present disclosure.

FIG. 4 is a flow diagram of an example process for training a cognitivecomputing system based on obtaining user-generated expanded conceptterms for particular seed concept terms.

FIG. 5 is a diagram of an example client computing device game GUI thatincludes a seed concept term and individual characters for use ingenerating expanded concept terms of the seed concept term, according toembodiments.

FIG. 6 is a diagram of an example client computing device game GUI thatincludes a seed concept term and individual characters for use ingenerating expanded concept terms of the seed concept term, according toembodiments.

FIG. 7 is a flow diagram of an example process for categorizing a firstset of user-generated concept terms and prompting each user to generatea second set of concept terms that are within the same category as theone or more the first concept terms, according to embodiments.

FIG. 8 is a diagram of an example client computing device game GUI thatincludes a user-generated concept term and individual characters for usein generating expanded concept terms of the user-generated concept term,according to embodiments.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to game-based training forcognitive computing systems. While the present disclosure is notnecessarily limited to such applications, various aspects of thedisclosure may be appreciated through a discussion of various examplesusing this context.

Cognitive computing systems may analyze terms and interpret the terms'meanings. For example, cognitive computing systems may perform conceptexpansion. Concept expansion is the process of inputting a set (i.e.,one or more) of seed concept terms that are expanded by the cognitivecomputing system to a more complete set of concept terms, which belongto the same category or semantic class. A “seed concept term” asdisclosed herein is a set of words, acronyms, or other statements thatmay be initially input into the cognitive computing system. A non-seedor expanded “concept term” as disclosed herein is an additional set ofwords, acronyms, or other statements that the cognitive computing systemor user inputs to expand on the original seed concept terms with a morecomplete set of concept terms. A “semantic class” may refer to a broadcategory, description, and/or meaning to which a concept term belongsto.

In an example illustration, a person may be interested in learning aboutmedicinal drugs that are related to the drugs he or she already knowsabout for remedial purposes. Accordingly, for the semantic class ofdrugs, the person may input into the concept expansion system a set of 3seed concept terms, “motrin, aspirin, and keflex.” The concept expansionsystem may then access from its information corpus (described morebelow) a set of conceptually related expanded concept terms in order toexpand the initial set of seed concept terms from 3 to the following 16concept terms: “allegra, lisinopril, metformin, equagesic, cimetidine,fiorinal, vancomycin, avelox, protinix, glimepiride, protonix,verapamil, norco, inderal, hctz, and advair.”

Concept expansion systems and cognitive computing systems in general maybe useful because human-input concept terms, whether seed concept termsor expanded concept terms may often be missing from dictionaries. Theseoften missed terms may include, for example, acronyms, abbreviations,spelling variants, informal shorthand terms (e.g., “abx” for“antibiotics”) and composite terms (e.g., “may-december,” or“virus/worm”). Further, new words may continually be developing, whichmay only be available on social medial portals. Concept expansion itselfmay be useful, for example, for search engines to collect large sets ofconcept terms to better interpret queries. Further, QA systems canutilize concept expansion to deal with list questions.

Cognitive computing systems may have to be trained before they are fullycapable simulating intelligence. Training is the process of buildingknowledge and learning a concept correctly or consistently. Currently,in order for a cognitive computing system to fully train, a subjectmatter expert (SME) may have to be utilized to make sure data returnedto users from the cognitive computing system is accurate or consistent.SMEs for cognitive computing are people that have expertise in aparticular domain that help train the cognitive computing systems withinthat domain, and help identify and upload content to form a knowledgebase (e.g., information corpus). A “domain” as disclosed herein is aparticular field of knowledge, specialty, industry, etc. (e.g.,medicine). In an example illustration, SMEs may create question andanswer pairs that form an answer key within a QA system.

Regarding concept expansion, SMEs may determine whether the cognitivecomputing system identifies correct expanded concept terms from the seedconcept terms. For example, the SME may first input a set of seedconcept terms to the cognitive computing system. The cognitive computingsystem may initially access and return 100 expanded concept terms. TheSME may then have to arduously parse through each of the 100 expandedconcept terms to determine the validity of the concept terms in relationto the seed concept terms. The SME may then only select a portion of the100 expanded concept terms (e.g., 20) that are valid concept terms. TheSME may then input the portion of valid concept terms into the cognitivecomputing system's knowledge repository (e.g., information corpus) suchthat when a user later inputs the same set of seed concept terms, thereturned expanded concept terms may reflect the portion of validconcepts that the SME selected during training. The other non-validconcept terms may be filtered out such that the cognitive computingsystem does not utilize the concept terms. The duties of SMEs maytherefore be very cumbersome and may waste valuable resources, such astime and money. Further, by only implementing one or a few SMEs,determining whether concept terms are valid may reflect a bias of thosefew SMEs during training. Therefore, embodiments of the presentdisclosure are directed to utilizing crowdsourcing and game-basedcognitive computing system training such that a select few SMEs do notengage in resource-consuming tasks and SMEs do not have to be heavilyrelied on. Further embodiments of the present disclosure are directed tocategorizing each of a first set of concept terms that a user generatesfor the game-based training and causing the client computing deviceassociated with the user to prompt the user to generate a second set ofconcept terms conceptually related to the first set of concept terms, asdescribed in more detail below.

As disclosed herein, to generate one or more concept terms that are“conceptually related” with seed concept terms means that concept termsand seed concept terms may be related by way of being synonyms,hypenyms, holonyms, hyponyms, merronyms, coordinate terms, verbparticles, troponyms, entailments, or any other types of beingassociated. As described herein, the term “characters,” may refer toletters, numbers, symbols, alpha-numeric characters, or any othersub-unit of a concept term or seed concept term.

FIG. 1 is a block diagram of an example computing environment 100,consistent with embodiments of the present disclosure. In someembodiments, the computing environment 100 may include one or moreremote devices 102, 112 (e.g., client computing devices) and one or morehost devices 122 (e.g., cognitive computing systems or server computingdevice). Remote devices 102, 112 and host device 122 may be distant fromeach other and communicate over a network 150 in which the host device122 comprises a central hub from which remote devices 102, 112 canestablish a communication connection. Alternatively, the host device andremote devices may be configured in any other suitable relationship(e.g., in a peer-to-peer or other relationship).

In some embodiments, the network 150 can be implemented by any number ofany suitable communications media (e.g., wide area network (WAN), localarea network (LAN), Internet, intranet, etc.). Alternatively, remotedevices 102, 112 and host devices 122 may be local to each other, andcommunicate via any appropriate local communication medium (e.g., localarea network (LAN), hardwire, wireless link, intranet, etc.). In someembodiments, the network 150 can be implemented within a cloud computingenvironment, or using one or more cloud computing services. Consistentwith various embodiments, a cloud computing environment may include anetwork-based, distributed data processing system that provides one ormore cloud computing services. Further, a cloud computing environmentmay include many computers, hundreds or thousands of them or more,disposed within one or more data centers and configured to shareresources over the network 150.

In some embodiments, host device 122 may include a natural languageprocessing (NLP) system 130, which is described in more detail below.The host device 122 may further include a user information database 134.The user information database 134 may include information about userssuch as identities of users, authentication information (e.g., usernameand password), game scores of particular users (discussed in more detailbelow), and subject matter expertise of users (e.g., title, careerspecialty, talents, etc.). Subject matter expertise of a user may beutilized by the host device 122 for determining a validity of conceptterms that a user generates when comparing a domain with the user'ssubject matter expertise, as described in more detail below.

In some embodiments, remote devices 102, 112 may enable users to submitquestions (e.g., search requests, potential research questions, or otheruser queries) to host devices 122 to retrieve search results. Forexample, the remote devices 102, 112 may include a query module 110, 120(e.g., in the form of a web browser or any other suitable softwaremodule) and present a graphical user interface (GUI) or other interface(e.g., command line prompts, menu screens, etc.) to solicit queries fromusers for submission to one or more host devices 122 and to displayanswers/results obtained from the host devices 122 in relation to suchuser queries.

Consistent with various embodiments, host device 122 and remote devices102, 112 may be computer systems, and may each be equipped with adisplay or monitor. The computer systems may include at least oneprocessor 106, 116, 126; memories 108, 118, 128; internal or externalnetwork interface or communications devices 104, 114, 124 (e.g., modem,network cards, etc.); optional input devices (e.g., a keyboard, mouse,or other input device); and any commercially available or customsoftware (e.g., browser software, communications software, serversoftware, NLP software, search engine and/or web crawling software,filter modules for filtering content based upon predefined criteria,etc.). In some embodiments, the computer systems may include servers,desktops, laptops, and hand-held devices (e.g., mobile phones, touchpads, smart watches, etc.).

Consistent with various embodiments, the remote devices 102 and 112 mayinclude respective gaming modules 136 and 138. In some embodiments, asillustrated in FIG. 1, the gaming modules 136 and 138 may becomputer-readable program instructions that are stored within thememories 108 and 118 respectively. In various embodiments, the gamingmodules 136 and/or 138 may be configured for providing (e.g.,displaying) a plurality of individual characters and one or more seedconcept terms and prompting a user of the respective remote devices 102and 112 to generate a set of one or more expanded concept terms that areconceptually related to the seed concept term, using one or more of theplurality of individual characters, as described in more detail below.Further the gaming modules 136 and 138 may be configured for providinggame scores to users of the remote devices 102 and 112, which correspondto point totals earned by the users for generating the expanded conceptterms. This and other functions for which the gaming modules 136 and 138are configured are explained in further detail below.

In various embodiments, the host device 122 may also include its owngaming module 140, which may be computer readable program instructionsstored in the memory 128. The gaming module 140, in some embodiments,may be configured for providing a seed concept term and a plurality ofindividual characters to each of the remote devices 102 and 112 (e.g.,cause the seed concept term and plurality of characters to bedisplayed), and prompting (e.g., causing a remote device to display anotification to) a user of the remote devices 102 and 112 to generate asecond set of one or more concept terms that are conceptually related tothe seed concept term, which is described in more detail below. Thegaming module 140 may further be configured for receiving a list ofexpanded concept terms generated by users of remote devices 102 and 112,and comparing the list of expanded concept terms with concept terms thehost device 122 has generated that are candidates for being conceptuallyrelated. The gaming module 140 may then determine whether any of theconcept terms that are candidates are included on the list and alter avalidity score of the candidates of expanded concept terms, or thoseexpanded concept terms that the users generated, based on thedetermining. These and other functions of the gaming module 140 aredescribed in more detail below.

In some embodiments, the host device 122 may be a server computingdevice that includes the gaming module 140 but not the natural languageprocessing system 130. In these embodiments, the server computing devicemay transmit a list of concept terms that a user has generated to acognitive computing system that includes the natural language processingsystem 130. Consistent with other embodiments, the host device 122 maybe a cognitive computing system that includes the natural languageprocessing system 130 but not the gaming module 140 such that the hostdevice 122 may be mainly responsible for performing concept expansion,QA generation, storing valid concept terms, etc. Accordingly, thecognitive computing system may be responsible for receiving informationfrom another server computing device, as described above. As illustratedin FIG. 1, in some embodiments, the host device 122 may be a servercomputing device and cognitive computing system such that the hostdevice 122 includes both the gaming module 140 and the natural languageprocessing system 130.

FIG. 2 is a block diagram of an example NLP system 212 usable togenerate answers to one or more questions, expand seed concept terms,etc., consistent with embodiments of the present disclosure. The NLPsystem 212 may be configured for generating a set of concept terms thatare candidates for being conceptually related to a given seed conceptterm. Aspects of FIG. 2 are directed toward an exemplary systemarchitecture 200, including a NLP system 212 to expand seed conceptterms. In some embodiments, one or more users can send requests forinformation to the NLP system 212 using a remote device (such as remotedevices 102, 112 of FIG. 1). Such a remote device may include a clientapplication 208 which may itself involve one or more entities operableto generate information that is then dispatched to system 212 vianetwork 215. NLP system 212 may be able to perform methods andtechniques for responding to the requests sent by the client application208. In some embodiments, the information received at NLP system 212 maycorrespond to input questions received from users, where the inputquestions may be expressed in a free form and in natural language. Insome embodiments, the information received at NLP system 212 maycorrespond to input seed concept terms received from users, or expandedconcept terms obtained from users.

A query or input (e.g., question) may be one or more words that form asearch term or request for data, information, or knowledge. The inputmay be expressed in the form of one or more keywords. Input may includevarious selection criteria and search terms. An input may be composed ofcomplex linguistic features in addition to keywords. However, akeyword-based search for answers to the inputs may also be possible. Insome embodiments, using restricted syntax for questions posed by usersmay be enabled. The use of restricted syntax may result in a variety ofalternative expressions that assist users in better stating their needs.

Consistent with various embodiments, client application 208 may operateon a variety of devices. Such devices may include, but are not limitedto, mobile and handheld devices (e.g., laptops, mobile phones, personalor enterprise digital assistants, and the like), personal computers,servers, or other computer systems that access the services andfunctionality provided by NLP system 212. In some embodiments, clientapplication 208 may include one or more components, such as a mobileclient 210. Mobile client 210, acting as an agent of client application208, may dispatch user query requests to NLP system 212.

Consistent with various embodiments, client application 208 may alsoinclude a search application 202, either as part of mobile client 210 orseparately, that may perform several functions, including some or all ofthe above functions of mobile client 210 listed above. For example, insome embodiments, search application 202 may dispatch requests forinformation to the NLP system 212. In some embodiments, searchapplication 202 may be a client application to NLP system 212. Searchapplication 202 may send requests for candidate answers to NLP system212. Search application 202 may be installed on a personal computer, aserver, or other computer system. In some embodiments, the mobile client208 may include a gaming GUI for use in prompting a user of the mobileclient 208 to input a set of seed concept term such that the NLP system212 may generate an expanded list of concept terms.

In some embodiments, search application 202 may include a search GUI 204and session manager 206. In such situations, users may be able to enterquestions or seed concept terms in search GUI 204. In some embodiments,search GUI 204 may be a search box or other GUI component, the contentof which can represent input to be submitted to NLP system 212. Usersmay authenticate to NLP system 212 via session manager 206. In someembodiments, session manager 206 may keep track of user activity acrosssessions of interaction with the NLP system 212. Session manager 206 mayalso keep track of what questions or concept terms are submitted withinthe lifecycle of a session of a user. For example, session manager 206may retain a succession of questions posed by a user during a session.In some embodiments, answers or concept expansion terms are produced byNLP system 212 in response to questions or seed concept terms.Information for sessions managed by session manager 206 may be sharedbetween computer systems and devices.

In some embodiments, client application 208 and NLP system 212 may becommunicatively coupled through network 215, e.g., the Internet,intranet, or other public or private computer network. In someembodiments, NLP system 212 and client application 208 may communicateby using Hypertext Transfer Protocol (HTTP), Representational StateTransfer (REST) calls, or any other suitable protocol. In someembodiments, NLP system 212 may reside on a server node. Clientapplication 208 may establish server-client communication with NLPsystem 212 or vice versa. In some embodiments, the network 215 can beimplemented within a cloud computing environment, or using one or morecloud computing services.

Consistent with various embodiments, NLP system 212 may respond to therequests for information sent by client applications 208 (e.g., seedconcept terms posed by users). NLP system 212 may generate expandedconcept terms according to the received seed concept terms, the semanticclass, and domain. In some embodiments, NLP system 212 may include ananalyzer 214, data sources 224, and a concept expander 228 (or answergenerator). Analyzer 214 may be a computer module (e.g., NaturalLanguage Processing (NLP) module) that analyzes the received questionsor concept terms. Analyzer 214 may perform various methods andtechniques for analyzing structured data (e.g., data from databases),unstructured data (e.g., data from a web page), and/or multimedia (e.g.,images, audio, video, etc.). For example, the question analyzer 214 mayutilize syntactic analysis and semantic analysis, as described below.

In some embodiments, analyzer 214 parses passages of documents. Analyzer214 may include various modules to perform analyses of receivedquestions or concept terms. For example, computer modules that analyzer214 may encompass, but are not limited to, may include a tokenizer 216,a part-of-speech (POS) tagger 218, a semantic relationship identifier220, and a syntactic relationship identifier 222.

In some embodiments, tokenizer 216 may be a computer module thatperforms lexical analysis. Tokenizer 216 may convert a sequence ofcharacters into a sequence of tokens. A token may be a string ofcharacters included in an electronic document and categorized as ameaningful symbol. Further, in some embodiments, tokenizer 216 mayidentify word boundaries in an electronic document and break any textpassages within the document into their component text elements, such aswords, multiword tokens, numbers, and punctuation marks. In someembodiments, tokenizer 216 may receive a string of characters, identifythe lexemes in the string, and categorize them into tokens.

Consistent with various embodiments, POS tagger 218 may be a computermodule that marks up a word in passages to correspond to a particularpart of speech and/or category. POS tagger 218 may read a passage orother text in natural language and assign a part of speech or categoryto each word or other token. POS tagger 218 may determine the part ofspeech to which a word (or other text element) corresponds based on thedefinition of the word and the context of the word. The context of aword may be based on its relationship with adjacent and related words ina phrase, sentence, question, or paragraph. In some embodiments, thecontext of a word may be dependent on one or more previously analyzedelectronic documents (e.g., the content of one source document may shedlight on the meaning of text elements in another source document).Examples of parts of speech that may be assigned to words include, butare not limited to, nouns, verbs, adjectives, adverbs, and the like.Examples of other part of speech categories that POS tagger 218 mayassign include, but are not limited to, comparative or superlativeadverbs, wh-adverbs, conjunctions, determiners, negative particles,possessive markers, prepositions, wh-pronouns, and the like. In someembodiments, POS tagger 218 may tag or otherwise annotate tokens of apassage with part of speech categories. In some embodiments, POS tagger218 may tag tokens or words of a passage to be parsed by the NLP system212.

In some embodiments, semantic relationship identifier 220 may be acomputer module that may identify semantic relationships and/or domainsof recognized text elements (e.g., words, phrases) in documents. In someembodiments, semantic relationship identifier 220 may determinefunctional dependencies between entities and other semanticrelationships. For example, when a user generates a set of seed conceptterms for the NLP system 212 to analyze (e.g., apple, orange, andcherry), the semantic relationship identifier 220 may first identify thesemantic class of the terms (e.g., fruit) to perform further analyses.

Consistent with various embodiments, syntactic relationship identifier222 may be a computer module that may identify syntactic relationshipsin a passage composed of tokens. Syntactic relationship identifier 222may determine the grammatical structure of sentences, for example, whichgroups of words are associated as phrases and which word is the subjector object of a verb. Syntactic relationship identifier 222 may conformto formal grammar.

In some embodiments, the analyzer 214 may be a computer module that canparse a received user query or concept terms and generate acorresponding data structure of the user query. For example, in responseto receiving a question at the NLP system 212, analyzer 214 may outputthe parsed question as a data structure. In some embodiments, the parsedquestion may be represented in the form of a parse tree or other graphstructure. To generate the parsed question, the analyzer 214 may triggercomputer modules 216, 218, 220, and 222. Additionally, in someembodiments, question analyzer 214 may use external computer systems fordedicated tasks that are part of the question parsing process.

In some embodiments, the concept expander 228 (or answer generator) maybe a computer module that generates expanded concept terms that arecandidates for being conceptually related to a seed concept term, orcandidate answers to posed questions. Examples of concept termsgenerated by concept expander 228 may include, but are not limited to,responses in the form of natural language sentences; reports, charts, orother analytic representation; raw data; web pages; and the like.

In some embodiments, the output of the analyzer 214 may be used bysearch application 202 to perform a search of a set of corpora toretrieve one or more expanded concept terms or candidate answerestimates to link to received questions. As used herein, a corpus mayrefer to one or more data sources. In an example illustration, if a userproposed a set a seed concept terms via the search application 202, thetarget documents within the corpora utilized to expand the seed conceptterms to other concept terms may include concept terms gathered fromsocial media, scientific articles, newspaper articles, books, videos,questionnaires, medical transcriptions, etc. For example, a user maydesire to expand upon his or her knowledge of U.S. presidents.Accordingly, the user may input the seed concept terms of Bush, Clinton,and Reagan. The search application 202 may then search within thecorpora (e.g., information corpus 226) and locate within variousuploaded electronic books to find the names of Lincoln, Washington, andother presidents. Accordingly, the NLP system 212 may return to the userthe expanded list of concept terms or additional names of Lincoln,Washington, and other presidents.

In some embodiments, data sources 224 may include data warehouses,information corpora, data models, multimedia, and document repositories.In some embodiments, the data source 224 may include an informationcorpus 226. The information corpus 226 may enable data storage andretrieval. In some embodiments, the information corpus 226 may be astorage mechanism that houses a standardized, consistent, clean andintegrated form of potential target documents (e.g., newspaper articles,published studies, books, etc.). The information corpus may store eachseed concept term and its associated expanded concepts. The data may besourced from various operational systems. Data stored in the informationcorpus 226 may be structured in a way to specifically address reportingand analytic requirements. In some embodiments, the information corpusmay be a relational database.

Consistent with various embodiments, concept expander 228 may includequery processor 230, visualization processor 232, and feedback handler234. When information in a data source 224 that matches a parsedquestion or expands upon an original seed concept term is located, atechnical query can be executed by query processor 230. Based on dataretrieved by a technical query executed by query processor 230,visualization processor 232 may be able to render visualization of theretrieved data, where the visualization represents the candidate answersor candidate expanded list of concept terms. In some embodiments,visualization processor 232 may render various analytics to representthe concept terms including, but not limited to, images, charts, tables,dashboards, maps, and the like. In some embodiments, visualizationprocessor 232 may present the expanded list of concept terms orcandidate answers to the user.

In some embodiments, feedback handler 234 may be a computer module thatprocesses feedback from users on expanded concept terms generated byconcept expander 228 or other users (e.g., the feedback handler 234implements training data). In some embodiments, users may generate datafor use by the NLP system 212 in order to determine which concept termsare valid (e.g., as provided via the gaming modules 136 and 138). Forexample, and as described in more detail below, a user on a remotedevice may be prompted to generate one or more concept terms that areconceptually related with a first identified seed concept term using aplurality of individual characters. A gaming module (e.g., gaming module140) may then obtain the list of the set of concept terms generated bythe user and determine whether each of the concept terms are valid suchthat they will be included in the corpora, as described in more detailbelow. The feedback handler 234 may then upload each of the valid wordsto the corpora.

FIG. 3 illustrates a block diagram of an example high level logicalarchitecture of a cognitive computing system, consistent withembodiments of the present disclosure. Aspects of FIG. 3 are directedtoward components and modules for use with a cognitive computing system.In some embodiments, host device 301 and remote device 302 may beembodied by host device 122 and remote device 102 of FIG. 1,respectively. In some embodiments, the analysis module 304, located onhost device 301, may receive a natural language seed concept term from aremote device 302, and can analyze the seed concept term to produceinformation about the seed concept term (e.g., provide expanded list ofcandidate concept terms). This may be accomplished, for example, byusing components 216, 218, 220, 222, and 240 of FIG. 2. The informationproduced by analysis module 304 may include, for example, the semanticclass of the seed concept term.

Next, the candidate generation module 306 may formulate queries (e.g.,seed concept terms) from the output of the analysis module 304 and thenpass these queries on to search module 308 which may consult variousresources (e.g., social media portals, blogs, books, studies, etc.) toretrieve documents that are relevant for providing a the list ofexpanded candidate concept terms. As used herein, documents may refer tovarious types of written, printed, or electronic media (includingpassages, web-pages, database files, multimedia, etc.) that provideinformation or evidence. As illustrated in FIG. 3, the search module 308may consult core information source 310. As used herein, a coreinformation source may refer to any document or group of documents thatis used by a relevant cognitive computing system to generate expandedconcept terms or generate candidate answers to user questions. Thecandidate generation module 306 may extract, from the search resultsobtained by search module 308, candidate expanded concept terms, whichit may then score (e.g., with confidence scores) and rank. A final listof expanded concept terms, based on a comparison of various confidencescores associated with the expanded concept terms, may then be sent fromthe candidate generation module 306 to remote device 302 forpresentation to the user. In addition, this information about expandedconcept terms and confidence scores may also be sent to informationsource quality control module 314. A user may respond, via remote device302, to generate conceptually related concept terms of a seed conceptterm (for example, the remote device 302 may receive user-generatedexpanded concept terms and upload the user-generated concept terms tothe cognitive computing system via a gaming module, as described below)through user feedback module 312. The user feedback module 312 may thenprovide this feedback to the information source quality control module314.

In some embodiments, the information source quality control module 314may compile and analyze information that it receives during the courseof normal operations of cognitive computing system 300. This receivedinformation (e.g., information from analysis module 304, candidategeneration module 306, and user feedback module 312) may be usable bythe information source quality control module 314 to determine whetherone or more new information sources should be ingested. When theinformation source quality control module 314 determines that a newinformation source having certain characteristics is needed (e.g., aninformation source that is associated with a specific user-generatedconcept term), it may instruct an ingestion module 316 accordingly.Based on these instructions, ingestion module 316 may search one or moreremote sources, such as remote corpora 318, in an attempt to locate oneor more suitable new information sources. In some embodiments, oncediscovered, these new information sources may be ingested by ingestionmodule 316 and become newly ingested information source 320. Thisinformation source may in turn be analyzed by training module 322. Thistraining analysis may take the form of training potential expandedconcept terms using the newly ingested information source 320 and thenreviewing the quality of the corresponding expanded concept terms. Insome embodiments, once a threshold level of confidence in the newinformation source is met, it may be combined with core informationsource 310 and used to generate expanded concept terms as users inputseed concept terms.

The various components and modules of the exemplary high level logicalarchitecture for a cognitive computing system described above may beused to implement various aspects of the present disclosure. Forexample, the analysis module 304 may, in some embodiments, be used toreceive a set of seed concepts from a user. The candidate generationmodule 306 and search module 308 may together, in some embodiments, beused to perform searches of core information source 310, generateexpanded concept terms, calculate confidence scores associated with theexpanded concept terms, and provide the expanded concept terms to one ormore users. Further, the information source quality control module 314may, in some embodiments, be used to analyze confidence scores anddetermine whether the confidence scores fail to meet one or moreconfidence criteria. Further, ingestion module 316 may, in someembodiments, be used to ingest new information sources (in response toan indication from the information source quality control module 314that a confidence criteria has not been satisfied).

FIG. 4 is a flow diagram of an example process 400 for training acognitive computing system based on obtaining user-generated expandedconcept terms for particular seed concept terms. It is to be understoodthat the order in which the blocks described below are discussed is notto be construed as limiting the order in which the individual acts maybe performed. In particular, the acts performed may be performedsimultaneously or in a different order than that discussed.

In some embodiments, the process 400 may begin at block 402 when thecognitive computing system identifies a plurality of seed concept terms.In some embodiments, the cognitive computing system may only identify asingle seed concept term. In other embodiments, a SME or other user mayidentify the seed concept terms. In an example illustration, thecognitive computing system may first identify a domain it will train in,such as orthopedic medicine. The cognitive computing system may thenidentify and select from its corpora (e.g., medical transcriptions datastores) a set of seed concept terms within the orthopedic medicinedomain. For example, the cognitive computing system may identify andselect the seed concept terms of inversion (type of body movement),arthroplasty (surgical procedure), spinal fusion (surgical procedure),and lordosis (inward curvature of spine).

Per block 404, the cognitive computing system may then select one of theseed concept terms as the first seed concept term. For example, usingthe illustration above, “inversion” may be selected. Per block 406, thecognitive computing system may then generate a first set of expandedconcept terms that are candidates for being conceptually related to thefirst seed concept term. To generate one or more concept terms that are“conceptually related” with seed concept terms means that concept termsand seed concept terms may be related by way of being synonyms,hypenyms, holonyms, hyponyms, merronyms, coordinate terms, verbparticles, troponyms, entailments, or by having any other association.For example, using the illustration above, if “inversion” (inwardmovement of a foot) was selected as the seed concept term to run conceptexpansion on, the cognitive computing system may then access from itscorpora (e.g., information corpus 226 of FIG. 2) synonyms of inversion(e.g., utilizing the analyzer 214 of FIG. 2) and provide the expandedconcept terms of: internal rotation (inward movement of arm/hip),abduction (outward movement of hip/arm), radial deviation (side movementof thumb towards wrist), and sacrum (triangular bone at end of externalspine that experiences little or no movement).

In some embodiments, per block 408, a server computing device (e.g.,cognitive computing system) or client computing device may parse each ofthe first set of concept terms into a plurality of individual characters(e.g., via the gaming module 140 of FIG. 1). For example, using theillustration above, the cognitive computing system may split internalrotation, abduction, radial deviation, and sacrum all into individualletters. The server computing device may shuffle or reorganize theplurality of individual characters prior to block 410 in order toconceal the words such that the user may find or generate the conceptterms individually on his or her own or even generate new concept terms,as described in more detail below. In some embodiments, the cognitivecomputing system may generate the first set of concept terms (block 406)and thereafter cause various client computing devices to display aplurality of individual characters without parsing the first set ofconcept terms into its letters. Rather, the cognitive computing systemmay provide a random generated set of letters to a client computingdevice in block 410. In some embodiments, a client computing device orother server computing system may be the entity that performs block 408instead of the cognitive computing system. In some embodiments, thecognitive computing system or server computing device may provide thefirst set of concept terms to the client computing device and the clientcomputing device performs block 408.

In some embodiments, per block 410, the server computing device mayprovide the plurality of individual characters and the seed concept termto users (e.g., by causing the individual characters and seed conceptterm to be displayed by the plurality of client computing devices, byproviding characters and seed concept term by audio, etc.). Inembodiments, the plurality of client computing devices may display theindividual characters and the seed concept term without the cognitivecomputing system or server computing device input (e.g., via the gamingmodule 136), as the client computing devices may be the entities thatidentify the selected seed concept term and provide the plurality ofindividual characters and the seed concept term. In some embodiments,block 410 may be a result of a service that has first solicited userconcept expansion as a free and downloadable game in order to recruit asmany participants as possible to generate concept terms from theplurality of individual characters, as described in more detail below.As described above, a potential advantage of soliciting the downloadablegame across the web is the idea of crowdsourcing to employ severalpeople from different perspectives and reducing resource costs (e.g.,SME workload) to choose which expanded concept terms or candidateexpanded terms initially generated by the cognitive computing systemsare valid.

Per block 412, and in some embodiments, the plurality of clientcomputing devices (or server computing device(s)) may prompt users ofthe plurality of computing devices to generate a second set of conceptterms that are conceptually related to the seed concept term by usingthe plurality of individual characters. Each user may therefore generatea second set of concept terms. Block 412 may correspond to the game thatis solicited to help train the cognitive computing system. For example,the game may be a word search game, wherein the user views the seedconcept term and tries to generate (e.g., come up with or create) wordsthat are related to the seed concept term. The layout, embodiments, andGUI of the game are described in more detail below. Using the exampleabove, the plurality of client computing devices (or server computingdevice(s)) may prompt each user to generate their own second set ofconcept terms that are conceptually related to the seed concept term of“inversion” by using each of the shuffled letters as found in theexpanded concept terms of internal rotation, abduction, radialdeviation, and sacrum.

Per block 414, the plurality of client computing devices (or servercomputing device(s)) may obtain respective lists (e.g., a compilation ofconcept terms) of the second set of concept terms from the plurality ofusers. In some embodiments, each of the users' lists are transmittedfrom a corresponding client computing device to the cognitive computingsystem (or other server computing device) for analysis. Alternatively,each client computing device may obtain the list and perform its ownanalysis (e.g., via the gaming module 136 of FIG. 1). Using theillustration above, after viewing the seed concept term of “inversion” afirst user, who is an orthopedic surgeon, may generate from theindividual characters the second concept terms of abduction, adduction(inward movement of hip/arm), internal rotation, external rotation(rotational outward movement of hip/arm), eversion (outward movement offoot), radial deviation, dorsiflexion (upward movement of foot), plantarflexion (downward movement of foot), and supination (type of inward footmovement). The orthopedic surgeon's list may then be transmitted to thecognitive computing system or be retained by the orthopedic surgeon'scomputing device for analysis.

Per block 416, the server computing device (e.g., cognitive computingsystem) may then determine (e.g., via the gaming module 140 of FIG. 1)if any of the first set of concept terms (that were generated by thecognitive computing system per block 406) are not included within any ofthe lists of the second set of concept terms (generated by the pluralityof users). Accordingly, the server computing device may compare thesecond sets of concept terms with the first set of concept terms todetermine if there is a match. Per block 420, if there are one or moreof the first concept terms that are not included on the lists of thesecond concept terms, then the server computing device may furtherreduce a validity score for the one of the first set of concept termsbased on the determining that the one or more of the first set ofconcept terms are not included on the list.

Invalidation, validation, validity, invalidity, etc. correspond to whatconcept terms the cognitive computing system will utilize or not utilizefor future concept expansion sessions or other analyses. These terms mayfurther correspond to determining a probability that an expanded conceptterm is correct in view the seed concept term. Using the exampleillustration above, if none of the users generated the concept term of“sacrum,” then the validity score for sacrum may be reduced such that“sacrum” may not be utilized for future concept expansion sessions forthe seed concept term “inversion”. Validity Score, validity scoring,etc. as described herein may mean to increase or decrease a value for,weight a value for, lower or increase the value of, change a ranking of,etc. of a particular concept term either generated by a user or thecognitive computing system towards or away from validity. For example,the concept term sacrum may start with a validity score of 100 anddecrease to a validity score of 60, for final validity score calculationpurposes as described in more detail below.

Per block 418, if all of the first set of concept terms are included inthe list of the second set of concept terms (e.g., there is a match),then the server computing device (or client computing device) mayincrease the validity score for each of the first set of concept terms.In some embodiments, block 418 may not be performed. Accordingly, block416 may only be utilized to reduce a validity score per block 420.

Per block 422, the server computing device may determine (e.g., via thegaming module 140 of FIG. 1) whether each term of the second sets ofconcept terms was generated by the plurality of users above a quantitythreshold. The cognitive computing system may then score each of thesecond set of concept terms accordingly for validation purposes. In someembodiments, the “quantity threshold” may correspond to a quantityfrequency or percentage (static or dynamic) at which the generation ofthe concept terms are acceptable for validation purposes. For example,using the illustration above, if the concept term “abduction” wasgenerated by 90% of the users (e.g., the static quantity threshold forvalidation may be 85%), then the server computing device may score“abduction” towards validity (e.g., increase a validity score).Alternatively, if the concept term abduction was generated over 50 timestotal by the plurality of users (e.g., the static quantity threshold maybe a raw number of 20), then abduction may also be scored towardsvalidity.

In some embodiments, the quantity threshold may be dynamic such that thecognitive computing system does not store a predetermined thresholdquantity requirement value but rather scores for validation a particularquantity of concept terms regardless of how frequently they weregenerated. For example, using the illustration above, because the seedconcept term “inversion” may be a domain-specific term, many peoplemight not understand the seed concept term in its proper context ordomain of orthopedic medicine. Accordingly, for example, if there wasnot one single expanded concept term that was generated by more than 5%of users, then the cognitive computing system may take this into accountand only retain for validation those concept terms that have beengenerated the most (e.g., retain the term that was generated by 5% ofthe users) because most people would not understand a particular conceptterm.

Per block 424, the server computing device (or a plurality of clientcomputing devices) may determine the subject matter expertise of eachuser according to the domain and weigh each of the user's generatedconcept terms accordingly for validation. In these embodiments, a domainmay first be identified that is associated with the seed concept term(e.g., orthopedic medicine). The server computing device (or clientcomputing device) may then identify a subject matter expertise of eachuser. For example, the client computing devices may require or ask usersbefore they play the concept term game to input their names,occupations, skills, knowledge, and/or know-how to identify a subjectmatter expertise of each user (which may be transmitted and stored tothe user information database 134 of FIG. 1). The server computingdevice (or client computing device) may then compare the subject matterexpertise of various users with the domain. The server computing device(or client computing device) may then provide a score, based on thecomparing, for validation purposes. For example, using the illustrationabove, the orthopedic surgeon's list of concept terms (abduction,adduction, internal rotation, external rotation, eversion, radialdeviation, dorsiflexion, plantar flexion, and supination) may each bescored higher (or weighed more strongly), as opposed to a list generatedby a computer technician who may not be as familiar with the domain oforthopedic medicine. Therefore, concept terms may be more likely to beutilized by the cognitive computing system or be valid if the person whogenerates the concept terms are familiar with the domain. In someembodiments, a domain may be identified and associated with a particularcategory, as opposed to a seed concept terms. Particular categories aredescribed in more detail below.

Per block 426, the server computing device may calculate final validityscore totals to determine which of the second set of concept terms arevalid (e.g., via the gaming module 140 of FIG. 1). In some embodiments,final validity score totals may be calculated by adding each of theindividual validity scores obtained from block 416, 422, and 424. Insome embodiments, more or less scores may be provided as illustrated inFIG. 4 for validity score totals. In some embodiments, final validityscore calculations may be static (e.g., simply adding each individualvalidity score). In other embodiments, calculations may be dynamic(e.g., determining that the subject matter expertise (block 424) of theorthopedic surgeon weighted or scored higher than any other scoreevaluations given the domain-specific seed concept term of “inversion”).In an example of a static final validity score calculation, using theillustration above for the concept term of “internal rotation,” thescore, per block 418, may increase towards validation since internalrotation may have been included in the first set of concept terms (e.g.,may increase by 5 points). Per block 422, internal rotation may havebeen the highest generated concept term (e.g., over 80% of usersgenerated the term), which may cause the validity score for “internalrotation” to increase by 15 points. Per block 424, each of the users whowere deemed to have appropriate subject matter expertise according tothe domain (e.g., physical therapists, orthopedic surgeons,chiropractors, etc.) may have generated the term internal rotation,which may cause a validity score increase of 20 points for that term.The server computing device (e.g., via the gaming module 140 of FIG. 1)may then add up each score (5+15+20) to arrive at a final validity scoreof 40.

Per block 428, the server computing device may then determine which ofthe first and second set of concept terms are valid. In someembodiments, a rank of the concept terms may be determined according tothe validity scores instead of determining validity. For example, theconcept terms of “abduction” may have the highest validity score andtherefore rank the highest (e.g., the cognitive computing system willprovide “abduction” as an expanded concept term of the seed term“inversion” with a high confidence interval or first on a list forfuture runs). In some embodiments, a concept term may only be valid ifit is over a threshold validation score total. For example, each conceptterm that has a total score over 50 may be determined to be valid.

Per block 430, the cognitive computing system may then be updated bystoring or retaining the valid concept terms to the corpora of thecognitive computing system. In some embodiments, only the valid conceptterms are used by the cognitive computing system. In other embodiments,all of the concept terms are utilized by the cognitive computing systembut in a ranked fashion according to the validity score. Using theillustration above, the server computing device may determine that theconcept terms of abduction, adduction, internal rotation, externalrotation, eversion, radial deviation, dorsiflexion, plantar flexion, andsupination are all valid concept terms. Consequently, a server computingdevice may transmit a list of the concept terms that are valid to thecognitive computing system. In some embodiments, the cognitive computingsystem is the server computing device and therefore does not have totransmit the list. The cognitive computing system may then identify andstore (e.g., via the feedback handler 234 of FIG. 2) the list to itscorpora for future concept expansion runs. For example, using theillustration above, if utilizing a concept expansion service, a usertyped in the concept seed term of “inversion,” the cognitive computingsystem may provide the updated list of expanded concept terms—abduction,adduction, internal rotation, external rotation, eversion, radialdeviation, dorsiflexion, plantar flexion, and supination—as opposed tothe original list of concept terms provided in block 406—internalrotation, abduction, and radial deviation. Therefore, the cognitivecomputing system may be efficiently trained by providing the game tomultiple users and scoring accordingly.

FIG. 5 is a diagram of an example client computing device game GUI thatincludes a seed concept term and individual characters for use ingenerating expanded concept terms of the seed concept term, according toembodiments. FIG. 5 includes a client computing device 500 (which mayalso be the remote devices 102 or 112 of FIG. 1), a GUI 502, whichincludes a header 502A that contains the seed concept term “hotel,” abackground 502B, and a plurality of individual characters 502C. In someembodiments, the GUI may be displayed via the gaming modules 136 and/or138 of FIG. 1. Consistent with embodiments, the plurality of individualcharacters 502C may include a two-dimensional array of cells (blocks)and each cell may include a letter. In some embodiments, the one or moreconcept terms may be generated by connecting one or more of therespective cells. For example, in some embodiments, 504A, 504B, 504C,504D, 504F, and 504E as represented by the arrows may each correspond tofinger gestures on a touch surface (e.g., touch pad, mobile phone, etc.)such that a user generates the expanded concept terms using the gesturesby connecting one or more of the individual characters 502C. In someembodiments, 504A, 504B, 504C, 504D, 504E, and 504F represent pointermovements or field sequences on a desktop computer or other device togenerate expanded concept terms by connecting one or more of theindividual characters 502C.

As illustrated in FIG. 5, the client computing device 500 may displaythe seed concept term of “hotel” within the header 502A. A user may beprompted to generate one or more concept terms that are associated withthe seed concept term of “hotel” using the one or more of the pluralityof individual characters 502C, such as performing a word search forwords conceptually related with the word “hotel”. As illustrated in FIG.5, the user may accordingly generate a list of four seed concept terms.For example, using the gesture 504A, the user may generate the conceptterm of “inn.” Using the gesture 504B, the user may generate the conceptterm of “cabin.” Using the gesture 504C, the user may generate theconcept term of “motel.” Using the gesture 504D, the user may generatethe concept term of “yurt.” Using the gesture 504E, the user maygenerate the concept term of “ORBITZ.” Using the gesture 504F, the usermay generate the concept term of “MARRIOTT.” These concept terms may allbe conceptually related (or be candidates for being conceptuallyrelated) with the seed concept term of “hotel.”

In some embodiments, the background 502B portion of the GUI 502 maydisplay a pictorial representation of a domain to a user. The domain maybe for use in providing a context for the seed concept term (orparticular category). Using the illustration of FIG. 5, for example, thebackground 502B may include a faint picture of a set of hotels,buildings, etc. This may help a user determine what context and field ofknowledge a particular seed concept term belongs to. In another example,if the seed concept term was “trunk,” the background 502B may include apicture of a tree in the background, as opposed to a car, to indicatethat the user should be generating concept terms based on a tree trunkinstead of a car trunk. In some embodiments, the background 502B mayinclude the areas just below and above the header 502A and the areas inbetween the cells of the individual characters 502C. In someembodiments, the server computing device may cause the client computingdevice 500 to display the pictorial representation.

In some embodiments, the seed concept term may be displayed within asentence to the client computing device 500 to indicate a domain orparticular category to which a particular concept terms belongs to. Thesentence may be displayed instead of or in addition to the pictorialrepresentation of the domain described above. Using the illustrationabove, for example, a header of the GUI may include the sentence “Treetrunks include bark,” with “tree trunk,” highlighted so as to indicatewhich words or phrases of the sentence are the seed concept terms. Thissentence may indicate the user should be generating concept terms basedon a tree trunk instead of a car trunk. In some embodiments, the servercomputing device may cause the client computing device 500 to displaythe sentence.

In some embodiments, each user may earn a particular amount of pointsfor generating each concept term and may accordingly accumulate a gamescore or point total earned at the end of a specified amount of time(e.g., 5 minutes). In some embodiments, as each user reaches a pointthreshold, the client computing device (or cognitive computing system)may provide additional characters (e.g., unlock new characters) suchthat a user can try to generate new concept terms. This may be for usein providing an incentive to generate as many concept terms as possible.Other incentive mechanisms may include providing or displaying a rankingbased on individual game scores such that a user may identify how wellhe or she performed when compared with other users. Further, the gamemay include a single player or multiple player mode such that each usermay choose how many users he or she will compete against.

FIG. 6 is a diagram of an example client computing device game GUI thatincludes a seed concept term and individual characters for use ingenerating expanded concept terms of the seed concept term, according toembodiments. FIG. 6 includes a computing device 600 and a GUI 602. TheGUI 602 includes the seed concept term of “hotel” and the variousexpanded concept terms of “mobile home,” “yurt,” “motel,” “cabin,” and“inn.” The GUI 602 may be provide via the gaming module 136 and/or 138of FIG. 1.

FIG. 6 illustrates that the game GUI 602 may not necessarily be a wordsearch game, but include other formats or rules such as the user maybuild their own words from individual characters. Accordingly, the gameformat may not include a predefined array of individual characters thatthe user must use to search for words (e.g., FIG. 5), but may include,for example, the seed word concept only and the user has to try and makea list of all of the concept terms associated with the seed wordconcept. In some embodiments, at least one character may be displayedsuch that a user may build his or her own words from the character. Forexample, using the illustration of FIG. 6, initially the GUI 602 mayonly include the seed concept term of “hotel,” and the letter “M” (or afirst concept term—e.g., mobile home). From the letter M, the user mayfirst generate the concept term of “mobile home” in a horizontal plane.Using the letter M, the user may also generate a concept term of “motel”in a vertical plane. Using the letter “B” within “mobile home,” the usermay generate the seed term “cabin.” Using the letter “I” in mobile home,the user may generate the concept term of “inn.” Using the concept termof “motel,” the user may generate the concept term of “yurt.” The gameGUI may be any suitable game format for generating various expandedconcept terms for given seed concept terms.

FIG. 7 is a flow diagram of an example process 700 for categorizing afirst set of user-generated concept terms and prompting each user togenerate a second set of concept terms that are within the same categoryas the one or more the first concept terms, according to embodiments. Insome embodiments, the process 700 may begin at block 702 when thecognitive computing system obtains a first list. The first list mayinclude a first set of one or more concept terms that a plurality ofusers generated. The first set of one or more concept terms may betransmitted from a plurality of client computing devices to thecognitive computing system (or server). In some embodiments, block 702may be block 414 as specified in FIG. 4. Accordingly, prior to block702, the cognitive computing system may have first generated its own setof concept terms that are candidates for being conceptually related to aseed concept term in some embodiments.

Further, prior to block 702, the plurality of client computing devicesmay provide (e.g., display) a plurality of individual characters and aseed concept term to a plurality of users using the respective pluralityof client computing devices. The plurality of client computing devicesmay prompt the plurality of respective users of the plurality of clientcomputing devices to generate the firsts set of one or more conceptterms that are conceptually related to the seed concept term using oneor more of the plurality of individual characters. Block 702 maytherefore be in response to the prompting of the plurality of respectiveusers to generate the first set of one or more concept terms that areconceptually related to the seed concept term. Although the process 700is illustrated as including a plurality of client computing devices anda plurality of respective users of the plurality of client computingdevices, it is understood that the process 700 may instead correspond toonly one user that utilizes only one client computing device.

Per block 704, the cognitive computing system may identify a firstdefinition (e.g., via the POS tagger 218 of FIG. 2) for each of thefirst set of one or more concept terms. A “definition” as describedherein may refer to a meaning of a particular concept term and/or seedconcept term. The definition may be derived from any suitable sourcesuch as a dictionary, thesaurus, book, article, etc. within a data store(e.g., Information Corpus 226 of FIG. 2). The definition may include apart of speech. A “part of speech” may describe a word type of a worddepending on the word's syntactic use or function in a sentence (e.g.,noun, adjective, pronoun, verb, adverb, preposition, conjunction,interjection, article, etc.). In an example illustration, a user maygenerate the concept term of “MARRIOTT” and the cognitive computingsystem may receive the concept term MARRIOTT and look up the definitionin a dictionary. The definition may read “the business and name of placeof lodging for a fee.” The dictionary may include a part of speechassociated with the definition of MARRIOTT, which may be “noun.” In someembodiments, a set of concept terms may include multiple definitions andmay therefore include different parts of speech. Accordingly, in someembodiments, the cognitive computing system may select a particulardefinition for a concept term based on ranking or scoring the differentdefinitions according to secondary factors. Such secondary factors mayinclude but are not limited to a particular domain that the set ofconcept terms belong to, comparing multiple concept terms together thata particular user has generated, etc. For example, for the concept termof “trunk,” the definition may correspond to a portion of a car orportion of a tree but if the domain is associated with a tree then thedefinition would be the portion of the tree. Therefore, the cognitivecomputing system may score the definition associated with a tree higherfor definition purposes.

In some embodiments, the cognitive computing system may utilize anontology (e.g., via the POS tagger 218) for a particular concept terminstead of or in addition to identifying a definition for thatparticular concept term. An “ontology” may be a particularconceptualization of a symbol (e.g., phrase, acronym, etc.) in aninformation system. Accordingly, an ontology may describe what a symbolrepresents as opposed to what a symbol means such that an entity mayunderstand the definition by a well-described or understood symbol.Ontologies may include one or more axioms (i.e., a rule or statementthat people generally regard as true) to better reflect a meaning of asymbol. The cognitive computing system may utilize ontologies by mappinga particular concept term to a symbol. For example, a symbol may be theword “Big Apple,” and particular concept terms that may be mapped to“Big Apple” are “large apples,” and/or “New York City.” The cognitivecomputing system may determine the ontology associations via variousdata sources within an information corpus such as social media,newspaper articles, etc. that have associated the concept terms with“Big Apple.”

Per block 706, the cognitive computing system may identify a seconddefinition (e.g., via the POS tagger 218) for a first seed concept term.For example using the illustration above, the seed concept term for theconcept term of MARRIOTT may be hotel. The definition of hotel may be “atype of place of lodging for a fee.”

Per block 708, the cognitive computing system may determine (e.g., viathe POS tagger), by at least comparing the first definition with thesecond definition, a relationship between the first seed concept termand each of the first set of one or more concept terms. Determining a“relationship” between the first seed concept term and each of the firstset of concept terms may include comparing words, phrases, and/or partsof speech of the first and second definitions to determine how similaror different the first seed concept term is from the first set of one ormore concept terms. Determining a relationship may also includecomparing passages (e.g., within the information corpus 1226 of FIG. 2)that the first seed concept term and each of the first concept terms arefound in. “Passages” may be part of an information source (e.g., socialmedia post) that may provide a context for what a particular seedconcept term or concept term means or is associated with. For example,the seed concept term of “hotel” may be selected from a blog within aninformation corpus in which someone stated in a passage: “hotels are theworst places to stay at.” The passage may therefore include thecorresponding sentence to which the seed concept term “hotel” belongsto. In some embodiments, ontologies may further be compared. In someembodiments, each of the first set of one or more concept terms may alsobe compared to each other in addition to comparing them with the firstseed concept term.

Utilizing the example illustration above, the first definition forMARRIOTT “the business and name of place of lodging for a fee,” and thesecond definition for hotel “a type of place of lodging for a fee” maybe compared. The cognitive computing system may, for example, identifythat MARRIOTT is a “name of a place of lodging . . . ” whereas “hotel”is “a type of place of lodging . . . ” which may be utilized tocategorize each concept term, as described in more detail below.

Per block 710, the cognitive computing system may select or determineone or more particular categories to which each of the first set of oneor more concept terms belongs to based on at least the determining ofthe relationship. A “category” or “subordinate category” as describedherein may be a distinct class that a particular set of concept termsbelongs to (which may be narrower than a domain or semantic class). Forexample, a domain may be “medicine” and a seed concept term may be“orthopedic surgery.” The seed concept term may be placed in thecategory of “orthopedics,” which is narrower than the domain ofmedicine. A category of various concept terms may have one or morecommon characteristics. For example, for the category of “orthopedics,”concept terms of “multiple sclerosis,” “angioplasty,” and “total kneereplacement,” have common characteristics in that each concept termeither refers to a musculoskeletal deformity or a correction of amusculoskeletal deformity. A category may also include an identification(e.g., orthopedics) that describes a specific meaning (or is a symbolthat represents) for a particular set of concept terms.

In an example of block 710 using the illustration above, becauseMARRIOTT is a “name of a place of lodging . . . ” whereas “hotel” is “atype of place of lodging . . . ” the cognitive computing system mayconsequently generate or place the seed concept term of “hotel” in thecategory of “type of lodging,” whereas the concept term of MARRIOTT maybe placed in the category of “name of lodging.”

Per block 712, and in some embodiments, the cognitive computing systemmay cause each of the plurality of client computing devices to display aplurality of individual characters, the first seed concept term, and aparticular concept term of the first set of one or more concept terms(e.g., FIG. 8) in order to indicate or hint at which category theparticular concept term belongs to. For example, if a client computingdevice displayed a seed concept term of “car,” and a generated conceptterm of “HONDA,” then it may be implicit that the user must generateconcept terms that are car manufacturers (e.g., TOYOTA, FORD, etc.).Alternatively, in some embodiments, the cognitive computing system maycause the plurality of client computing devices to each explicitlydisplay a category that the particular concept term belongs to. Forexample, a GUI of each client computing device may explicitly displaythe plurality of individual characters, the particular concept term andthe category to which the concept term belongs to.

In some embodiments, block 712 may be in response to the cognitivecomputing system causing each of the plurality of client computingdevices to display the one or more particular categories to which thefirst set of one or more concept terms belongs that each user generated.Each of the client computing devices may then receive a request for aparticular concept term that belongs to a category. For example, a GUIof a client computing device may display each category and under eachcategory each of the concept terms in which a particular user generatedmay be listed under each category such that it is evident whatparticular category each generated concept term belongs to. The clientcomputing device may then prompt (e.g., notify via a message) therespective user to select one of the first set of one or more conceptterms (in order to generate a second set of one or more concept termsthat are within the same category as one of the first set of one or moreconcept terms, as described in more detail below). The client computingdevice may then receive a user request for the particular concept termand the client computing device may then, in response the receiving ofthe request, display the plurality of individual characters and theparticular concept term (e.g., as illustrated in FIG. 8).

In some embodiments, each of the client computing devices may display anotification that specifies or instructs that the user must generateconcept terms that are within the same category as the particularconcept terms in order to earn points. After the notification, then onlythe particular concept term may be displayed along with the individualcharacters (e.g., without the seed concept term or the category to whichthe particular concept term belongs to). Although embodiments herein aredescribed as “displaying” a plurality of characters, concept terms,etc., it is recognized that methods other than displaying may beutilized to provide data. For example, the client computing device mayprovide the concept terms via voice activation command such that a usermay hear the concept terms as opposed to reading the concept term.

Per block 714, all of the client computing devices may prompt respectiveusers of the client computing devices to generate a second set ofconcept terms that are within the same category as the particularconcept term of the first set using the plurality of individualcharacters. In some embodiments, the prompting in block 714 may includea notification to each user that the user should generate concept termsthat are part of the same category and will be awarded pointsaccordingly for generated concept terms within the same category. Forexample, if one of the first concept terms was “hotel,” the category maybe “a type of lodging”. Accordingly, a user may receive points forgenerating concept terms that fall in the same category (e.g., motel,yurt, cabin), whereas the user may not receive points (or receive lesspoints) for generating concept terms outside of the category (e.g.,MARRIOTT) even though they may be within the same domain or beconceptually related to an associated seed concept term. In someembodiments, block 714 may not occur at all. In some embodiments, theclient computing devices may prompt users to generate a second set ofconcept terms that are conceptually related to the particular conceptterm. In some embodiments, the prompting in block 714 may be thedisplaying or providing of various indicia as specified in block 712such that not explicit notification is given. For example, by displayingthe plurality of individual characters, the seed concept term, and aparticular concept term that a user generated, then this may besufficient enough information to “prompt” each user to generate a secondset of concept terms.

Per block 716, the cognitive computing system may obtain a second list.The second list may include a second set of one or more concept termsgenerated by users that are transmitted from the plurality of clientcomputing devices to the cognitive computing system (or server). Wheneach user generates a subset or portion of the second set of conceptterms, the respective client computing device may transmit each of theportion to the cognitive computing system.

Per block 718, the cognitive computing system may cause each of theclient computing devices to provide a game score (a point total earned)for respective users that have generated one or more of the second setof concept terms. In some embodiments, the cognitive computing systemmay not necessarily cause the client computing devices to provide a gamescore, but the client computing devices gaming module may provide thegame score on its own. In some embodiments, each user will earn highergame scores for generating concept terms that are part of the samecategory as the particular concept term.

Per block 720, the cognitive computing system may determine whether anyof the users generated concept terms (first or second set of one or moreconcept terms) that were part of a new category. A “new” category may bea category that the cognitive computing system (e.g., the POS tagger218) has not yet generated. For example, referring back to FIG. 4, thecognitive computing system, per block 406, may first generate a firstset of concept terms that are associated with a seed concept terms(i.e., perform concept expansion) in order to train. In doing conceptexpansion, the cognitive computing system may have generated its owncategories for a set of concept terms that are candidates for beingconceptually related to the seed concept term. Accordingly, the when thecognitive computing system receives the plurality of lists from theplurality of client computing devices, the cognitive computing systemmay or may not have already generated a category for a particular set ofconcept terms. In various embodiments, the decision in block 720 may beas a result of users generating either the first set of concept termsand/or the second set of concept terms. For example, block 720 may beplaced immediately below block 702 when the cognitive computing systemobtains a plurality of lists of a first set of one or more concept termsfrom a plurality of client computing devices.

Per block 722, if any of the users generated concept terms that are partof a new category then the cognitive computing system may cause theclient computing device(s) to increase the game score for thoserespective users that generated concept term(s) that are part of a newcategory or sub-category. This may incentivize users to generate as manydifferent types of concept terms as possible. Generating as manydifferent types of concept terms as possible may in turn help thecognitive computing system have a more robust concept expansion systemsuch that any given set of seed concept terms may be expanded to severalconcept terms that would have otherwise not be available withoutincentivizing users to create as many categories as possible.

Per block 724, the cognitive computing system or other server computingdevice may determine whether each of the second set of concept termswere generated above a quantity threshold and score each concept termaccordingly for validation. Block 724 may be analogous to block 422 ofFIG. 4. Per block 726 the cognitive computing system or other servercomputing device may determine the subject matter expertise of each useraccording to the domain and score each of the user's generated conceptterms accordingly for validation. Block 726 may be analogous to block424 of FIG. 4. In some embodiments, the cognitive computing system maydetermine whether any concept terms that have already been generated bythe cognitive computing system were not included in the second set ofone or more concept terms that the users have generated. This may beanalogous to block 416 of FIG. 4. If there were some concept terms thatwere generated by the cognitive computing system but which were notgenerated by any of the client computing devices, then the cognitivecomputing system or server computing device may reduce a validity scorefor such concept terms.

Per block 728, the cognitive computing system or server computing devicemay calculate final validity score totals, which is analogous to block426 of FIG. 4. In some embodiments, the final validity score totals maybe based solely on the second set of concept terms that were generatedand whether the second set is with a particular category. In someembodiments, the final validity score total may include concept termsthat were generated by users that are candidates for being conceptuallyrelated to a seed concept term (e.g., FIG. 4). Per block 730, thecognitive computing system or server computing device may determinewhich of the first and second set of concept terms are valid (which mayinclude the first and second set of concept terms as specified in FIG.4). Per block 732, the cognitive computing system may update itself bystoring or retaining the valid concept terms to the corpus of thecognitive computing system (analogous to block 430 of FIG. 4). Block 732may be performed to train the cognitive computing system for futureconcept expansion runs.

FIG. 8 is a diagram of an example client computing device game GUI thatincludes a user-generated concept term and individual characters for usein generating expanded concept terms of the user-generated concept term,according to embodiments. The client computing device 800 may include aheader 802A and within the header 802A the user-generated concept termof MARRIOTT 802A1 may be displayed as well as the seed concept term ofhotel 802A2. In some embodiments, the seed concept term of hotel 802A2is not displayed or provided.

In some embodiments, FIG. 8 may be an extension of the game GUI asillustrated in FIG. 5 for a particular user. For example, as illustratedin FIG. 5 a user may be prompted to generate a set of concept terms thatare candidates for being conceptually related to the seed concept termof “hotel.” The user may accordingly generate the concept terms of“inn,” “cabin,” “motel,” “yurt,” “MARRIOTT,” and “ORBITZ.” In responseto the user generating these concept terms, the client computing device500 may transmit (e.g., via the gaming module 136 of FIG. 1) each of theconcept terms to a cognitive computing system wherein the cognitivecomputing system categorizes (e.g., via the POS tagger 218) each of theconcept terms as described above. For example, the cognitive computingsystem may categorize each of the concept terms of “inn,” “cabin,”“motel,” and “yurt” as a “type of lodging,” whereas the cognitivecomputing system may categorize the concept term of “MARRIOTT” as a“name of a place of lodging.” The cognitive computing system may furthercategorize the concept term of ORBITZ as a “service for bookingtravel-related items” (which includes hotels). Therefore, each of thegenerated concept terms may belong to three categories.

In some embodiments, in response to the cognitive computing systemcategorizing the concept terms, the cognitive computing system may thentransmit each of the identified categories and concept terms back to theclient computing device. For example, using the illustration above, thecognitive computing system may transmit the three categories and each ofthe associated concept terms (e.g., inn, cabin, motel, yurt, MARRIOTT,and ORBITZ) back to the client computing device 500. In response to thetransmitting, a GUI of the client computing device 500, for example, maydisplay (e.g., via the gaming module 136) each of the three categoriesand under each category each of the concept terms in which the usergenerated may be listed under each category such that it is evident whatparticular category each generated concept term belongs to.

As described above, the client computing device 500 may then prompt(e.g., notify via a message) a user to select a concept term that he orshe generated in order to generate another set of concept terms that arewithin the same category as (or conceptually related to) the selectedconcept term. The client computing device 500 may then receive a userrequest for a particular concept term and the client computing device500 may then, in response to the receiving of the request, display theplurality of individual characters and the particular selected conceptterm. For example, the client computing device 800 may be the clientcomputing device 500 at a second subsequent time after the user hasgenerated concept terms that are candidates for being conceptuallyrelated to the seed concept term of “hotel.” Therefore, the user mayhave selected the concept term of “MARRIOTT,” which may have beengenerated via the gesture 504F at the first time according to FIG. 5. Atthe second time, the computing device 800 may then display at least theconcept term of MARRIOTT 602A1 and the plurality of individualcharacters 802C. The client computing device 800 may then prompt theuser to generate a second set of concept terms that are conceptuallyrelated to the selected concept term of MARRIOTT 802A1 in which the usergenerated at the first time according to FIG. 5. In some embodiments,the client computing device 800 may prompt the user to generate thesecond set of concept terms that are not only conceptually related tothe concept term of MARRIOTT 602A1, but are within the same category asMARRIOTT (i.e., a name of a place of lodging). Accordingly, the user maygenerate the concept terms of HILTON via the gesture 604A and COURTYARDvia the gesture 604B. HILTON and COURTYARD may also be associated withparticular businesses that are names of places of lodging. In someembodiments, as described above, a user may not necessarily utilize handgestures on a touch surface to generate concept terms but rather utilizepointers (e.g., a mouse) or keyboard controls to generate concept terms.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer-implemented method, comprising:obtaining, by a cognitive computing system including a natural languageprocessing system, a first list that includes a first set of one or moreconcept terms from an information corpora of the cognitive computingsystem, the first set of one or more concept terms are transmitted froma first client computing device to the cognitive computing system, thefirst set of one or more concept terms are candidates for beingconceptually related to a seed concept term, wherein prior to theobtaining, the first client computing device provides a first pluralityof individual characters and the seed concept term to a first user usingthe first client computing device, the first client computing deviceprompts the first user to generate a second set of one or more conceptterms that are conceptually related to the seed concept term using oneor more of the first plurality of individual characters; determining, bythe natural language processing system and in response to the obtaining,one or more particular categories to which each of the first set of oneor more concept terms belongs, wherein each of the particular categoriesis a distinct class that includes an identification, the identificationdescribes a specific meaning for each of the first set of one or moreconcept terms; associating, in response to the determining that one ormore particular categories to which each of the first set of one or moreconcept terms belongs, at least one of the second set of one or moreconcept terms with the one or more particular categories of the firstset of one or more concept terms; causing, in response to theassociating the second set of one or more concept terms with the one ormore particular categories of the first set of one or more conceptterms, the client computing device to increase a game score for thefirst user by a first amount, the game score corresponding to a pointtotal earned by the first user for generating the second set of one ormore concept terms; determining that at least one of the particularcategories is a new category that the cognitive computing system has notyet generated; associating, in response to the determining that at leastone of the particular categories is a new category, at least one of thesecond set of one or more concept terms with the new category; causing,in response to the associating at least one of the second set of one ormore concept terms with the new category, the client computing device tofurther increase the game score for the first user by a second amountthat is different than the first amount; and training the cognitivecomputing system by updating the first set of one or more concept termsin the information corpora to include one or more of the second set ofone or more concept terms that the cognitive computing system can usefor a subsequent execution of the method.
 2. The method of claim 1,wherein the determining one or more particular categories includes:identifying a first definition for each of the first set of one or moreconcept terms, the first definition including a first part of speech;identifying a second definition for the seed concept term, the seconddefinition including a second part of speech; determining, by at leastcomparing the first definition with the second definition, arelationship between the seed concept and each of the first set of oneor more concepts; and determining the one or more particular categoriesbased on at least the determining a relationship.
 3. The method of claim1, further comprising: causing, by the cognitive computing system, thefirst client computing device to provide a second plurality ofindividual characters; causing, by the cognitive computing system, thefirst client computing device to provide one of the first set of conceptterms; and causing, by the cognitive computing system, the first clientcomputing device to provide the seed concept term, wherein the firstclient computing device prompts the first user to generate a second setof one or more concept terms that are within a same category as the oneof the first set of concept terms term using one or more of the secondplurality of individual characters.
 4. The method of claim 3, furthercomprising: obtaining, by the cognitive computing system, a second listfrom the first client computing device, the second list including thesecond set of one or more concept terms; identifying, by the cognitivecomputing system, a domain associated with the one or more particularcategories, the domain corresponding to a field of knowledge;identifying, by the cognitive computing system, a subject matterexpertise of the first user; comparing, by the cognitive computingsystem, the subject matter expertise of the first user with the domain;and providing a validity score for the second set of one or more conceptterms based on the comparing the subject matter expertise of the firstuser with the domain, the validity score corresponding to scoring thesecond set of one or more concept terms to determine whether the secondset of one or more concept terms are utilized by the cognitive computingsystem for concept expansion, and wherein concept expansion is a processof inputting a set of seed concept terms that are expanded by thecognitive computing system to a more complete set of concept terms. 5.The method of claim 3, further comprising: causing, by the cognitivecomputing system and prior to the causing the first client computingdevice to provide a second plurality of individual characters, the firstclient computing device to display the one or more particular categoriesto which each of the first set of one or more concept terms belongs; andcausing, by the cognitive computing system and prior to the causing thefirst client computing device to provide a second plurality ofindividual characters, the first client computing device to display theeach of the first set of one or more concept terms.
 6. The method ofclaim 1, further comprising: determining that the second set of one ormore concept terms does not include all of the first set of one or moreconcept terms; causing, in response to the determining that the secondset of one or more concept terms does not include all of the first setof one or more concept terms, the client computing device to decreasethe game score for the first user.
 7. A system comprising: a computingdevice having a processor; and a computer readable storage medium havingprogram instructions embodied therewith, the program instructionsexecutable by the processor to cause the system to: display a firstplurality of individual characters; display a first seed concept term,the first seed concept term to train a cognitive computing systemincluding a natural language processing system, wherein the cognitivecomputing system analyzes the first seed concept term to generate afirst set of one or more concept terms from an information corpora ofthe cognitive computing system that are candidates for beingconceptually related to the first seed concept term; display aparticular concept term of a second set of one or more concept terms,wherein the particular concept term is generated by a first user, theparticular concept term is assigned to a particular category by thenatural language processing system, wherein the particular category is adistinct class that includes an identification, the identificationdescribes a specific meaning for the particular concept term; prompt thefirst user to generate a third set of one or more concept terms that arewithin the particular category as the particular concept term using oneor more of the first plurality of individual characters; provide a gamescore to the first user, the game score corresponding to a point totalearned by the first user for generating the third set of concept terms;increasing the game score for each of the second set of concept termsthat is included in the third set of concept terms by a first pointamount; increasing the game score for each of the third set of conceptterms that is not included in the second set of concept terms by asecond point amount that is different than the first point amount; andtrain the cognitive computing system by updating the first set of one ormore concept terms in the information corpora to include one or more ofthe second set of one or more concept terms that the cognitive computingsystem can use for a subsequent execution of the program instructions.8. The system of claim 7, wherein the program instructions executable bythe processor further cause the system to obtain a list of the third setof concept terms.
 9. The system of claim 8, wherein the programinstructions executable by the processor further cause the system totransmit the list of the third set of concept terms to the cognitivecomputing system, wherein the cognitive computing system determines, bycomparing the first set of concept terms with the third set of conceptterms, that one of the first set of concept terms is not included on thelist, the cognitive computing system further providing a validity scorefor the third set of concept terms based on the determining that one ofthe first set of concept terms is not included on the list.
 10. Thesystem of claim 7, wherein the program instructions executable by theprocessor further cause the system to display a pictorial representationof a domain, the domain for use in providing a context for theparticular category.
 11. The system of claim 7, wherein the programinstructions executable by the processor further cause the system todisplay the particular concept term within a sentence to indicate adomain, the domain for use in providing a context for the particularcategory.
 12. The system of claim 7, wherein the first plurality ofindividual characters are displayed within in a two-dimensional array ofrespective cells, the third set of concept terms each being generated byconnecting one or more of the respective cells.
 13. The system of claim7, wherein the program instructions executable by the processor furthercause the system to decrease the game score for each of the second setof concept terms that is not included in the third set of concept termsby a third point amount.
 14. A computer program product comprising acomputer readable storage medium having program instructions embodiedtherewith, the program instructions executable by a cognitive computingsystem including a natural language processing system to cause thecognitive computing system to perform a method, the method comprising:obtaining a first list that includes a first set of one or more conceptterms from an information corpora of the cognitive computing system, thefirst set of one or more concept terms are transmitted from a pluralityof client computing devices, the first set of one or more concept termsare candidates for being conceptually related to a seed concept term,wherein prior to the obtaining, the plurality of client computingdevices provide a first plurality of individual characters and a seedconcept term to a plurality of respective users of the plurality ofclient computing devices, the plurality of client computing devicesprompt the plurality of respective users to generate the first set ofone or more concept terms that are conceptually related to the seedconcept term using one or more of the first plurality of individualcharacters; determining, in response to the obtaining and by the naturallanguage processing system, one or more particular categories to whicheach of the first set of one or more concept terms belongs, wherein eachof the particular categories is a distinct class that includes anidentification, the identification describes a specific meaning for eachof the first set of one or more concept terms; associating, in responseto the determining that one or more particular categories to which eachof the first set of one or more concept terms belongs, at least one ofthe second set of one or more concept terms with the one or moreparticular categories of the first set of one or more concept terms;causing, in response to the associating the second set of one or moreconcept terms with the one or more particular categories of the firstset of one or more concept terms, the client computing device toincrease a game score for the first user by a first amount, the gamescore corresponding to a point total earned by the first user forgenerating the second set of one or more concept terms; determining thatat least one of the particular categories is a new category that thecognitive computing system has not yet generated; associating, inresponse to the determining that at least one of the particularcategories is a new category, at least one of the second set of one ormore concept terms with the new category; causing, in response to theassociating at least one of the second set of one or more concept termswith the new category, the client computing device to further increasethe game score for the first user by a second amount that is differentthan the first amount; and training the cognitive computing system byupdating the first set of one or more concept terms in the informationcorpora to include one or more of the second set of one or more conceptterms that the cognitive computing system can use for a subsequentexecution of the method.
 15. The computer program product of claim 14,wherein the determining one or more particular categories includes:identifying a first definition for each of the first set of one or moreconcept terms, the first definition including a first part of speech;identifying a second definition for the seed concept term, the seconddefinition including a second part of speech; determining, by at leastcomparing the first definition with the second definition, arelationship between the seed concept and each of the first set of oneor more concepts; and determining the one or more particular categoriesbased on at least the determining a relationship.
 16. The computerprogram product of claim 14, wherein the method further comprises:causing the plurality of client computing devices to provide a secondplurality of individual characters; causing the plurality of clientcomputing devices to provide a first concept term of the first set ofone or more concept terms that each respective user of the plurality ofrespective users has generated; and causing the plurality of clientcomputing device to provide the seed concept term, wherein the pluralityof client computing devices prompt the respective users to generate asecond set of one or more concept terms that are within a same categoryas the first concept term using one or more of the second plurality ofindividual characters.
 17. The computer program product of claim 16,wherein the method further comprises: obtaining a second list, thesecond list including the second set of one or more concept terms. 18.The computer program product of claim 17, wherein the method furthercomprises: determining that a particular concept term of the second setof one or more concept terms was generated by two or more of theplurality of respective users above a quantity threshold; in response tothe determining, increasing a validity score for the particular conceptterm, wherein the validity score corresponds to scoring the particularconcept term to determine whether the particular concept term isutilized by the cognitive computing system for concept expansion, andwherein concept expansion is a process of inputting a set of seedconcept terms that are expanded by the cognitive computing system to amore complete set of concept terms which belongs to a semantic class asthe set of seed concept terms.
 19. The computer program product of claim17, wherein the method further comprises: identifying a domainassociated with one of the one or more particular categories, the domaincorresponding to a field of knowledge; identifying a subject matterexpertise of a first user of the plurality of respective users, whereinthe first user generates a particular concept term of the second set ofconcept terms that is within the one of the one or more particularcategories; comparing the subject matter expertise of the first userwith the domain; and providing a validity score for the particularconcept term based on the comparing the subject matter expertise of thefirst user with the domain, the validity score corresponding to scoringthe particular concept term to determine whether the particular conceptterm is utilized by the cognitive computing system for conceptexpansion, and wherein concept expansion is a process of inputting a setof seed concept terms that are expanded by the cognitive computingsystem to a more complete set of concept terms.
 20. The computer programproduct of claim 14, wherein the method further comprises: determiningthat the second set of one or more concept terms does not include all ofthe first set of one or more concept terms; causing, in response to thedetermining that the second set of one or more concept terms does notinclude all of the first set of one or more concept terms, the clientcomputing device to decrease the game score for the first user.